Skip to content

feat(relay): make Postgres pool size configurable, default 50 - #3191

Merged
tlongwell-block merged 1 commit into
mainfrom
eva/db-pool-size-50
Jul 28, 2026
Merged

feat(relay): make Postgres pool size configurable, default 50#3191
tlongwell-block merged 1 commit into
mainfrom
eva/db-pool-size-50

Conversation

@tlongwell-block

Copy link
Copy Markdown
Collaborator

Summary

  • Raise the relay's Postgres pool cap from the buzz-db default of 20 to 50 per pool, and expose BUZZ_DB_POOL_SIZE for per-deploy tuning
  • Applies to the writer pool and, when READ_DATABASE_URL is set, the reader pool; zero/unparsable values fall back to the default
  • The buzz-db library default is unchanged — only the relay opts into the larger cap

Why

During the 2026-07-27 18:40–19:05Z traffic burst on bb-public, per-pod PG pools pinned at 20 fleet-wide and ~380 requests failed on the 3s acquire timeout — membership checks, channel access lookups, and historical queries returning errors to users. The database was nowhere near a limit: Aurora (db.r8g.8xlarge, ~5,000 max connections) sat at 19% CPU, 201 connections (~4% of capacity), commit latency flat at 0.01ms.

The 20-connection default was sized for "four relay pods against PG max_connections=100" (the comment in buzz-db says exactly that). Production now runs 12–15 pods against Aurora — the per-pod cap is the binding constraint, not the DB.

Budget at the new default: 15 pods × (50 writer + 50 reader + 5 audit) ≈ 1,575 potential connections, ~30% of Aurora's ceiling — and actual usage stays demand-driven (min_connections stays 2, connections only open under load).

Same shape as #2521 (BUZZ_REDIS_POOL_SIZE), which fixed the identical class of ceiling on the Redis side.

Testing

  • cargo test -p buzz-relay: 762 passed, 1 failed — the lone red is api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo, the known pre-existing flake; it fails identically on clean main at the same SHA (verified via git stash / rerun)
  • New test db_pool_size_env_override_and_invalid_fallback covers override, zero, and unparsable fallback
  • defaults_are_valid extended to pin the new default
  • cargo clippy -p buzz-relay --all-targets -- -D warnings and cargo fmt --check clean

The buzz-db default of 20 connections per pool was sized months ago
for a handful of relay pods against PG max_connections=100. Production
now runs 12+ pods against Aurora (~5,000 connections), where that cap
is the binding constraint: during the 2026-07-27 18:40Z traffic burst,
per-pod pools pinned at 20 fleet-wide and ~380 requests failed on the
3s acquire timeout while the database sat at 19% CPU and ~4% of its
connection capacity.

Raise the relay's default to 50 and expose BUZZ_DB_POOL_SIZE for
per-deploy tuning. The cap applies to the writer pool and, when
READ_DATABASE_URL is set, the reader pool. Zero or unparsable values
fall back to the default. The buzz-db library default is unchanged.

At 15 pods x (50 writer + 50 reader + 5 audit) = ~1,575 potential
connections, ~30% of Aurora's ceiling; actual usage stays demand-driven
(min_connections=2).

Co-authored-by: Tyler Longwell <tlongwell@block.xyz>
Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
@tlongwell-block
tlongwell-block requested a review from a team as a code owner July 27, 2026 19:42
@tlongwell-block
tlongwell-block merged commit 2ce2d71 into main Jul 28, 2026
32 checks passed
@tlongwell-block
tlongwell-block deleted the eva/db-pool-size-50 branch July 28, 2026 13:17
tlongwell-block pushed a commit that referenced this pull request Jul 28, 2026
* origin/main:
  feat(relay): make Postgres pool size configurable, default 50 (#3191)

Co-authored-by: Tyler Longwell <tlongwell@block.xyz>
Signed-off-by: Tyler Longwell <tlongwell@block.xyz>

# Conflicts:
#	crates/buzz-relay/src/main.rs
tlongwell-block pushed a commit that referenced this pull request Jul 28, 2026
* origin/main: (70 commits)
  feat(relay): make Postgres pool size configurable, default 50 (#3191)
  Publish symbol-bearing debug relay images (#3250)
  feat(tracing): add datastore tracing plumbing (#2760)
  fix(buzz-acp): accept id-keyed config options when resolving model switch (#2795)
  fix(desktop): probe legacy Goose install dir on Windows (#3248)
  refactor(desktop): extract install command execution into install_exec (#3251)
  Polish composer activity layout and transitions (#3151)
  feat(invites): add use-limited invite links (#3141)
  fix(node): bump Buzz-supplied Node runtimes past OpenClaw's >=24.15.0 floor (#3218)
  fix(desktop): preserve thread anchor through layout reflow (#3212)
  feat(search): parse from:/in:/after:/before: and pass them in the filter (#2871)
  fix(desktop): fetch join policies through native networking (#2862)
  fix(desktop): republish agent identity records when a persona rename propagates (#2607)
  fix(desktop): keep project Inbox previews compact (#3193)
  Inbox refactor (#2045)
  Fix composer selection formatting and drop overlay (#3172)
  Refine pending message status (#3153)
  feat(admin): show reported message content in report detail (#3149)
  fix(desktop): recover full local storage on startup (#3182)
  Replace mobile reconnect banners with skeleton shimmer (#3143)
  ...

Co-authored-by: Tyler Longwell <tlongwell@block.xyz>
Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
cameronhotchkies pushed a commit to cameronhotchkies/buzz that referenced this pull request Jul 28, 2026
…y-to-env

* origin/main: (22 commits)
  Polish mobile message and search layouts (block#3121)
  Add mobile message image galleries (block#3312)
  chore(release): release Buzz Desktop version 0.5.0 (block#3213)
  feat(relay): make Postgres pool size configurable, default 50 (block#3191)
  Publish symbol-bearing debug relay images (block#3250)
  feat(tracing): add datastore tracing plumbing (block#2760)
  fix(buzz-acp): accept id-keyed config options when resolving model switch (block#2795)
  fix(desktop): probe legacy Goose install dir on Windows (block#3248)
  refactor(desktop): extract install command execution into install_exec (block#3251)
  Polish composer activity layout and transitions (block#3151)
  feat(invites): add use-limited invite links (block#3141)
  fix(node): bump Buzz-supplied Node runtimes past OpenClaw's >=24.15.0 floor (block#3218)
  fix(desktop): preserve thread anchor through layout reflow (block#3212)
  feat(search): parse from:/in:/after:/before: and pass them in the filter (block#2871)
  fix(desktop): fetch join policies through native networking (block#2862)
  fix(desktop): republish agent identity records when a persona rename propagates (block#2607)
  fix(desktop): keep project Inbox previews compact (block#3193)
  Inbox refactor (block#2045)
  Fix composer selection formatting and drop overlay (block#3172)
  Refine pending message status (block#3153)
  ...

Signed-off-by: npub1x4hk035p3p9q39a3fcrd2fe30lpkrhr5dwe0cqzzjphxyyh8m0gsq4vqap <356f67c681884a0897b14e06d527317fc361dc746bb2fc0042906e6212e7dbd1@buzz.block.builderlab.xyz>
Co-authored-by: npub1x4hk035p3p9q39a3fcrd2fe30lpkrhr5dwe0cqzzjphxyyh8m0gsq4vqap <356f67c681884a0897b14e06d527317fc361dc746bb2fc0042906e6212e7dbd1@buzz.block.builderlab.xyz>
calvadev pushed a commit to shopstr-eng/buzz that referenced this pull request Aug 3, 2026
…3191)

## Summary

- Raise the relay's Postgres pool cap from the `buzz-db` default of 20
to 50 per pool, and expose `BUZZ_DB_POOL_SIZE` for per-deploy tuning
- Applies to the writer pool and, when `READ_DATABASE_URL` is set, the
reader pool; zero/unparsable values fall back to the default
- The `buzz-db` library default is unchanged — only the relay opts into
the larger cap

## Why

During the 2026-07-27 18:40–19:05Z traffic burst on bb-public, per-pod
PG pools pinned at 20 fleet-wide and ~380 requests failed on the 3s
acquire timeout — membership checks, channel access lookups, and
historical queries returning errors to users. The database was nowhere
near a limit: Aurora (db.r8g.8xlarge, ~5,000 max connections) sat at 19%
CPU, 201 connections (~4% of capacity), commit latency flat at 0.01ms.

The 20-connection default was sized for "four relay pods against PG
max_connections=100" (the comment in `buzz-db` says exactly that).
Production now runs 12–15 pods against Aurora — the per-pod cap is the
binding constraint, not the DB.

Budget at the new default: 15 pods × (50 writer + 50 reader + 5 audit) ≈
1,575 potential connections, ~30% of Aurora's ceiling — and actual usage
stays demand-driven (`min_connections` stays 2, connections only open
under load).

Same shape as block#2521 (`BUZZ_REDIS_POOL_SIZE`), which fixed the identical
class of ceiling on the Redis side.

## Testing

- `cargo test -p buzz-relay`: 762 passed, 1 failed — the lone red is
`api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo`, the
known pre-existing flake; it fails identically on clean `main` at the
same SHA (verified via `git stash` / rerun)
- New test `db_pool_size_env_override_and_invalid_fallback` covers
override, zero, and unparsable fallback
- `defaults_are_valid` extended to pin the new default
- `cargo clippy -p buzz-relay --all-targets -- -D warnings` and `cargo
fmt --check` clean

Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Co-authored-by: Tyler Longwell <tlongwell@block.xyz>
wpfleger96 added a commit that referenced this pull request Aug 8, 2026
## Buzz Relay release v0.2.1

### Changes since relay-v0.2.0:

- fix(sdk): preserve self-mention p tags in message and forum event
builders ([#4975](#4975))
([`78c87ae20e`](78c87ae))
- feat(desktop): adding rich link previews to messages
([#3818](#3818))
([`1922d49cb2`](1922d49))
- feat(relay): accept kind:30179 private managed-agent events at ingest
([#5133](#5133))
([`ad923353a2`](ad92335))
- fix(media): require authenticated reads
([#4610](#4610))
([`769ac70b74`](769ac70))
- feat(identity): recover desktop identity from a signed-in phone
([#4845](#4845))
([`6eb65919f1`](6eb6591))
- ci: prove the relay-driven mesh lifecycle — discover, join, infer,
deny — with real nodes
([#3862](#3862))
([`38bf642fcf`](38bf642))
- relay: fuzz WebSocket 1012 restart-close timing on graceful drain
(BUZZ_DRAIN_JITTER_MS)
([#4542](#4542))
([`e14fff74d0`](e14fff7))
- fix(reactions): support max-length custom emoji
([#3833](#3833))
([`2ea9385015`](2ea9385))
- fix(channels): restrict private-channel invitations
([#4612](#4612))
([`efe1893dd3`](efe1893))
- fix(workflow): bind trigger author to the signed event
([#4607](#4607))
([`885bed35ee`](885bed3))
- fix(git): revoke access for banned relay members
([#4608](#4608))
([`997b8caaa4`](997b8ca))
- Define private managed agent wire protocol
([#4593](#4593))
([`067c085f37`](067c085))
- perf(relay): index channel-id lookups and skip trace-only reads
([#4647](#4647))
([`bc9e6528a7`](bc9e652))
- Polish mobile inbox and media flows
([#4512](#4512))
([`feccf4eabc`](feccf4e))
- fix(git): allow deleting the default branch
([#4297](#4297))
([`fc598f5f8d`](fc598f5))
- feat(projects): add buzz projects CLI commands (NIP-MP kind:30621)
([#4020](#4020))
([`b7bb15122e`](b7bb151))
- perf(relay): serve relay-membership checks from the read replica
([#4124](#4124))
([`ac4fa13b8e`](ac4fa13))
- fix(relay): allow open relays to set their NIP-11 workspace icon
(kind:9033) ([#3998](#3998))
([`5765fc74b7`](5765fc7))
- feat(relay): accept kind:30621 multi-repo projects at ingest
([#3171](#3171))
([`cb9701cd30`](cb9701c))
- feat(relay): raise hosted community limit to five
([#3829](#3829))
([`10d5a26414`](10d5a26))
- fix(relay): align NIP-11 max_limit with REQ ceiling
([#3635](#3635))
([`23f0c26b1c`](23f0c26))
- feat(relay): gate kind 30178 team-catalog reads behind the shared tag
([#3358](#3358))
([`114d40d9d3`](114d40d))
- fix(db): isolate usage metrics advisory-lock test on scratch DB
([#3670](#3670))
([`dba97eecd9`](dba97ee))
- perf(presence): reduce heartbeat frequency
([#3783](#3783))
([`bf139e8d0b`](bf139e8))
- feat(mesh): upgrade embedded mesh to v0.74 and harden shared compute
(split 1/2 of #3467) ([#3741](#3741))
([`4933672eb4`](4933672))
- feat(replica): portable heartbeat-token fence with snapshot-local
reader routing ([#3268](#3268))
([`63496cc1d4`](63496cc))
- fix(git): channel binding tooling + author remediation for unbound
repos ([#3626](#3626))
([`788b3c002b`](788b3c0))
- feat: configure S3 URL addressing style
([#3400](#3400))
([`7012d86d52`](7012d86))
- feat(tracing): correlate trace IDs in relay logs
([#3608](#3608))
([`005b5b819a`](005b5b8))
- fix(relay): avoid subscription lock inversion
([#3413](#3413))
([`22be8bb351`](22be8bb))
- feat(cli): add users set-status command for NIP-38 profile status
([#3253](#3253))
([`60158fce3e`](60158fc))
- feat(relay): make Postgres pool size configurable, default 50
([#3191](#3191))
([`2ce2d71cc3`](2ce2d71))
- feat(tracing): add datastore tracing plumbing
([#2760](#2760))
([`e94b9aeda0`](e94b9ae))
- feat(invites): add use-limited invite links
([#3141](#3141))
([`d500c2d5cf`](d500c2d))
- feat(admin): show reported message content in report detail
([#3149](#3149))
([`f069a85503`](f069a85))
- resolve findings ([#3150](#3150))
([`9b0f744804`](9b0f744))
- Revert "fix(cli,relay): resolve agents by verified owner"
([#3168](#3168))
([`a041e2d21e`](a041e2d))
- fix(cli,relay): resolve agents by verified owner
([#2615](#2615))
([`c3084b36d9`](c3084b3))
- fix(security): enforce durable community ban on NIP-43 relay-admin
kinds 9030-9033 ([#3128](#3128))
([`e2e0079101`](e2e0079))
- fix(security): authorize kind:9000 role changes in both directions
([#3017](#3017))
([`00ecf2cac7`](00ecf2c))
- feat(desktop): handle project work from Inbox
([#3117](#3117))
([`c5c4f390b6`](c5c4f39))
- feat(relay): make per-owner community limit configurable via
BUZZ_MAX_COMMUNITIES_PER_OWNER
([#2599](#2599))
([`2a051a404d`](2a051a4))
- feat(relay): add author-only-unless-shared read gate for kind 30175
([#2768](#2768))
([`ab3af82871`](ab3af82))
- fix(core): block IPv6 transition SSRF targets
([#2801](#2801))
([`c26bf5945d`](c26bf59))
- fix(workflow): bypass system proxies for webhooks
([#2800](#2800))
([`60a171b19e`](60a171b))
- fix(audit): hash created_at at the precision Postgres stores
([#2638](#2638))
([`264a56a226`](264a56a))
- feat(desktop): make pull request reviews actionable
([#2510](#2510))
([`9081ab0ec9`](9081ab0))
- fix(relay): decompress gzip-encoded git smart-HTTP request bodies
([#2670](#2670))
([`5ca36e7b91`](5ca36e7))
- fix(sharing): preserve agent/team snapshot tEXt chunks through media
sanitization ([#2438](#2438))
([`b096b0a15a`](b096b0a))
- fix(relay): send 1012 restart close to all clients on graceful drain
([#2575](#2575))
([`1911c69aa2`](1911c69))
- fix(media): sanitize animated image uploads
([#2524](#2524))
([`8f8f5fa5a4`](8f8f5fa))
- fix(channels): strip leading hash prefixes from names
([#2250](#2250))
([`d0ab3fdb05`](d0ab3fd))
- feat(relay): make Redis pool size configurable, default 16
([#2521](#2521))
([`bcc3e13069`](bcc3e13))
- feat(desktop+acp): spawn a harness per (agent, community) pair at GUI
startup — warm sockets, lazy LLM pool
([#2122](#2122))
([`61cc738ee8`](61cc738))
- feat(media): add S3-truth per-community storage sweep
([#2044](#2044))
([`bd37a4d584`](bd37a4d))
- feat(relay): log NIP-98 pubkey attribution on HTTP bridge requests
([#2206](#2206))
([`7e34bee62c`](7e34bee))
- Revert "feat(relay): inventory unreachable Git objects"
([#2275](#2275))
([`0fb820f9bf`](0fb820f))
- feat(relay): inventory unreachable Git objects
([#2264](#2264))
([`3afc9dae15`](3afc9da))
- relay: add author_type label to buzz_events_stored_total
([#2243](#2243))
([`b9f54c43fe`](b9f54c4))
- fix(git): make project branch workflows reliable
([#2213](#2213))
([`166f27be4b`](166f27b))
- feat(cli): manage repository protection rules
([#2193](#2193))
([`f94324598d`](f943245))
- feat(cli): add agents archive/unarchive/archived subcommands
([#2173](#2173))
([`7d7992067b`](7d79920))
- fix(mobile): sanitize Android image uploads
([#2188](#2188))
([`ee21da90bd`](ee21da9))
- fix(cli): paginate channel directory queries
([#2181](#2181))
([`03fe19d603`](03fe19d))
- fix(mobile): image upload fails due to unstripped metadata
([#2185](#2185))
([`37f15b2001`](37f15b2))
- perf(relay): compact Git packs before manifest limits
([#2172](#2172))
([`80e0ab16b0`](80e0ab1))
- perf(relay): cache Git pack hydration
([#2169](#2169))
([`a4d82ec722`](a4d82ec))
- fix(relay): bound and observe Git read operations
([#2167](#2167))
([`5f7c93d9c1`](5f7c93d))
- relay: gate push enqueue on live leases; batch matcher pipeline
(T1b/T1a-repair/T2b) ([#2145](#2145))
([`e43b2d5aac`](e43b2d5))
- relay: add audit logging disable switch
([#2134](#2134))
([`bf5acabdde`](bf5acab))
- relay: skip TTL deadline bump for known-permanent channels (T1a
write-amp) ([#2125](#2125))
([`2e936d439c`](2e936d4))
- fix(git): carry NIP-OA delegation in auth event
([#2120](#2120))
([`c12257d57a`](c12257d))
- Route lag-tolerant reads to an optional Postgres read replica
([#2084](#2084))
([`29c48883d3`](29c4888))
- fix: recover community access visibility
([#2074](#2074))
([`ca384d082d`](ca384d0))
- feat: proxy feedback-scoped admin attachments
([#2059](#2059))
([`d7f918e3cb`](d7f918e))
- feat: add read-only deployment moderation dashboard
([#1999](#1999))
([`68e670e001`](68e670e))
- Bug-bash round 2: table scroll, Goose instructions, workflow mention
wake ([#2034](#2034))
([`64b8fea6dc`](64b8fea))
- Strip media metadata on clients and reject it at the relay
([#2006](#2006))
([`5cfd69cb0c`](5cfd69c))
- [codex] Hold Git concurrency permits through streaming (BUZZ-SEC-018)
([#1916](#1916))
([`7baea42abb`](7baea42))
- [codex] Enforce shared relay admission limits (BUZZ-SEC-019)
([#1917](#1917))
([`73fc0ec6cf`](73fc0ec))
- [codex] Block banned actors from moderation commands (BUZZ-SEC-007)
([#1915](#1915))
([`caa195ca58`](caa195c))
- [codex] Fix relay WebSocket admission limits
([#1682](#1682))
([`d3ce971fc7`](d3ce971))
- feat: add invite QR and mobile direct join
([#1957](#1957))
([`648cbf3610`](648cbf3))
- fix(join-policy): require legal consent on hosted invites
([#1987](#1987))
([`2e1577f76f`](2e1577f))
- [codex] Prevent actor-tag UI impersonation
([#1931](#1931))
([`c540ec9678`](c540ec9))
- Scope relay runtime state by community
([#1658](#1658))
([`d52dedb06f`](d52dedb))
- Apply optional relay join policy across join flows
([#1894](#1894))
([`6c2d667575`](6c2d667))
- feat(media): require auth for relay media reads
([#1926](#1926))
([`f308762852`](f308762))
- feat(relay): add community unarchive endpoint
([#1908](#1908))
([`6b9641db2b`](6b9641d))
- feat(relay): gate Git web GUI separately
([#1901](#1901))
([`34dc7dec75`](34dc7de))
- mesh: upgrade runtime, enforce membership, add shared compute provider
([#1656](#1656))
([`54638ff4bb`](54638ff))
- Route Git scratch through configured volume
([#1884](#1884))
([`2318b3096c`](2318b30))
- feat(relay): gate usage metrics behind stable leader
([#1814](#1814))
([`59e9821503`](59e9821))
- Relay mesh: cross-pod tunnel + huddle transport (buzz-relay-mesh)
([#1670](#1670))
([`ccb021d713`](ccb021d))
- feat(push): deliver accepted relay events as wakes
([#1866](#1866))
([`bffbc5f22c`](bffbc5f))
- fix(db): resolve duplicate migration version
([#1863](#1863))
([`08ad38a07f`](08ad38a))
- Add private product feedback sidecar
([#1857](#1857))
([`af190c93e1`](af190c9))
- feat(relay): add durable community archival
([#1834](#1834))
([`2b15a72675`](2b15a72))
- feat(push): add public APNs gateway
([#1770](#1770))
([`1c006822e4`](1c00682))
- feat(relay): add atomic community ownership transfer
([#1845](#1845))
([`52e42ccb9f`](52e42cc))
- Bound NIP-RS retention and search indexing
([#1771](#1771))
([`1b4703021d`](1b47030))
- Add optional standalone pairing relay to Helm chart
([#1799](#1799))
([`9b47c8548f`](9b47c85))
- fix(relay): publish membership snapshot on provisioning
([#1761](#1761))
([`0950d392b7`](0950d39))
- feat(relay): per-community usage metrics
([#1723](#1723))
([`620822899a`](6208228))
- refactor(desktop): remove vestigial MCP toolsets config
([#1776](#1776))
([`dfec75b3c0`](dfec75b))

**To release:** merge this PR. The tag and build will happen
automatically.

Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants